from matplotlib import pyplot as plt
import numpy as np
import pandas as pd
import geopandas as gpd
np.random.seed(42)

Lecture 13A: From Notebooks to the Web: Github Pages, Web Servers, and Dash

November 24, 2020

Housekeeping

• Assigment #7 (including final project proposal) due on Thursday 12/3
• There will be additional office hours the week of Dec 14th (the week after classes end)

Final project due: 5pm on Monday, December 21st

Outline for the rest of the course

We'll discuss ways to translate our analysis results to the Web in a meaningful way. We'll cover three methods to do so, each with their own pros and cons:

1. Embedding interactive charts on the Web via Github Pages
2. Creating Web apps & dashboards using the Python library Dash
3. Creating Web apps & dashboards using the Python library Panel

Today: we'll focus on the first two today and cover Panel in detail in week 14

Part 1: Embedding Interactive Charts on the Web

To start, let's recap three ways that we've learned to produce interactive charts in the course:

• Altair
• Hvplot
• Folium

Example 1: Measles Incidence in Altair

Load the data from week 2:

url = "https://raw.githubusercontent.com/MUSA-550-Fall-2020/week-2/master/data/measles_incidence.csv"
data = pd.read_csv(url, skiprows=2, na_values="-")

data.head()

	YEAR	WEEK	ALABAMA	ALASKA	ARIZONA	ARKANSAS	CALIFORNIA	COLORADO	CONNECTICUT	DELAWARE	...	SOUTH DAKOTA	TENNESSEE	TEXAS	UTAH	VERMONT	VIRGINIA	WASHINGTON	WEST VIRGINIA	WISCONSIN	WYOMING
0	1928	1	3.67	NaN	1.90	4.11	1.38	8.38	4.50	8.58	...	5.69	22.03	1.18	0.4	0.28	NaN	14.83	3.36	1.54	0.91
1	1928	2	6.25	NaN	6.40	9.91	1.80	6.02	9.00	7.30	...	6.57	16.96	0.63	NaN	0.56	NaN	17.34	4.19	0.96	NaN
2	1928	3	7.95	NaN	4.50	11.15	1.31	2.86	8.81	15.88	...	2.04	24.66	0.62	0.2	1.12	NaN	15.67	4.19	4.79	1.36
3	1928	4	12.58	NaN	1.90	13.75	1.87	13.71	10.40	4.29	...	2.19	18.86	0.37	0.2	6.70	NaN	12.77	4.66	1.64	3.64
4	1928	5	8.03	NaN	0.47	20.79	2.38	5.13	16.80	5.58	...	3.94	20.05	1.57	0.4	6.70	NaN	18.83	7.37	2.91	0.91

5 rows × 53 columns

Use the pandas.melt() function to convert it to tidy format:

annual = data.drop('WEEK', axis=1)
measles = annual.groupby('YEAR').sum().reset_index()
measles = measles.melt(id_vars='YEAR', var_name='state', value_name='incidence')

measles.head()

	YEAR	state	incidence
0	1928	ALABAMA	334.99
1	1929	ALABAMA	111.93
2	1930	ALABAMA	157.00
3	1931	ALABAMA	337.29
4	1932	ALABAMA	10.21

Now let's load altair:

import altair as alt

# use a custom color map
colormap = alt.Scale(
    domain=[0, 100, 200, 300, 1000, 3000],
    range=[
        "#F0F8FF",
        "cornflowerblue",
        "mediumseagreen",
        "#FFEE00",
        "darkorange",
        "firebrick",
    ],
    type="sqrt",
)

# Vertical line for vaccination year
threshold = pd.DataFrame([{"threshold": 1963}])

# plot YEAR vs state, colored by incidence
chart = (
    alt.Chart(measles)
    .mark_rect()
    .encode(
        x=alt.X("YEAR:O", axis=alt.Axis(title=None, ticks=False)),
        y=alt.Y("state:N", axis=alt.Axis(title=None, ticks=False)),
        color=alt.Color("incidence:Q", sort="ascending", scale=colormap, legend=None),
        tooltip=["state", "YEAR", "incidence"],
    )
    .properties(width=650, height=500)
)

rule = alt.Chart(threshold).mark_rule(strokeWidth=4).encode(x="threshold:O")

out = chart + rule
out

Saving Altair Plots

Altair plots can be fully represented as JSON data. This makes them very easy to embed on websites, as we shall soon see!

# save the chart as JSON
out.save("measlesAltair.json") 

# save the chart as HTML
out.save("measlesAltair.html")

Now, let's compare the HTML and JSON files...

Measles Incidence in hvplot

import hvplot.pandas
import hvplot

# Make the heatmap with hvplot
heatmap = measles.hvplot.heatmap(
    x="YEAR",
    y="state",
    C="incidence", # color each square by the incidence
    reduce_function=np.sum, # sum the incidence for each state/year
    frame_height=450,
    frame_width=600,
    flip_yaxis=True,
    rot=90,
    colorbar=False,
    cmap="viridis",
    xlabel="",
    ylabel="",
)

# Some additional formatting using holoviews 
# For more info: http://holoviews.org/user_guide/Customizing_Plots.html
heatmap = heatmap.redim(state="State", YEAR="Year")
heatmap = heatmap.opts(fontsize={"xticks": 0, "yticks": 6}, toolbar="above")
heatmap

type(heatmap)

holoviews.element.raster.HeatMap

Saving Hvplot Plots

HTML is are only option here...

import hvplot

hvplot.save(heatmap, 'measlesHvplot.html')

Example 3: Folium + OSMnx

• This is a recap of the material we covered in Week 9.
• We'll quickly walk through how we used osmnx to created an interactive Folium map of the shortest route between two points.

import osmnx as ox

Identify the lat/lng coordinates for our places of interest: Use osmnx to download the geometries for the Libery Bell and Art Museum

philly_tourism = ox.geometries_from_place("Philadelphia, PA", tags={"tourism": True})

art_museum = philly_tourism.query("name == 'Philadelphia Museum of Art'").squeeze()

art_museum.geometry

liberty_bell = philly_tourism.query("name == 'Liberty Bell'").squeeze()

liberty_bell.geometry

# Liberty Bell coords
liberty_bell_coords = (liberty_bell.geometry.y, liberty_bell.geometry.x)

# Art Musuem coords
art_museum_coords = (art_museum.geometry.centroid.y, art_museum.geometry.centroid.x)

Get the street graph in Center City: Use osmnx to download the street network around City Hall.

G = ox.graph_from_address('City Hall, Philadelphia, USA', 
                          dist=1500, 
                          network_type='drive')

Identify the nodes in the graph closest to our points of interest.

# Get the origin node
orig_node = ox.get_nearest_node(G, liberty_bell_coords) 

# Get the destination node
dest_node = ox.get_nearest_node(G, art_museum_coords) 

Use networkx to find the shortest path

import networkx as nx

# Calculate the shortest path between these nodes
route = nx.shortest_path(G, orig_node, dest_node)

# The underlying graph of streets
graph_map = ox.plot_graph_folium(G, popup_attribute='name', edge_width=2)

# Plot the route with folium on top of the previously created graph_map
route_graph_map = ox.plot_route_folium(G, route, route_map=graph_map)

Saving Folium Maps

Just use the save() function!

route_graph_map.save("foliumChart.html")

from IPython.display import IFrame

# Display the HTML file in the notebook
IFrame("foliumChart.html", width=600, height=500)

How can we display these charts on the Web?

We can embed them on GitHub Pages...

Part 2: Github Pages

• We can embed these interactive charts on static (no server) websites such as Github Pages
• Github Pages is static website that can be automatically generated from any Github repository

Template Site with Examples

• I've created a template repository that you can use to quickly get up and running with Github Pages
• Available at: https://github.com/MUSA-550-Fall-2020/github-pages-starter
• The rendered site is available at: https://musa-550-fall-2020.github.io/github-pages-starter

Github Pages URL

The structure of the URL for the rendered page is:

https://[USERNAME].github.io/[REPOSITORY NAME]

Note: you can create a new website for every repository, so this will work even if you have a personal Github Pages website set up.

Step 1: Create your own repository

• The main repository at https://github.com/MUSA-550-Fall-2020/github-pages-starter is a template repository
• Click on the "Use this template" button to create your own version of the repository

For more information, see this guide on creating a repository from a template.

Step 2: Choose a name and description

Step 3: Enable Github Pages on your new repository

On the home page for your new repository go to "Settings":

• Scroll down, and under the "Github Pages" section, select the "Master" as the branch to be used for GitHub pages.
• This will automatically build the website at the following URL: https://[USERNAME].github.io/[REPOSITORY NAME]

Step 4: Customize your site

• Enter your site name, description, etc by editing the _config.yml file.
• There are 3 different ways that you can make changes to your blog's files:
  • Edit files within your in the browser at GitHub.com
  • Use a third party GitHub content editor, like Prose by Development Seed.
  • Clone down your repository and make updates locally, then push them to your GitHub repository.

For more information on the config file, see the documentation.

Step 5: Publish a post

• Posts are written in Markdown and any files added to the _posts directory will be automatically uploaded and published
• Useful resource: Markdown Cheatsheet

Adding new posts

To add new posts, simply add a file in the _posts directory that:

1. follows the convention YYYY-MM-DD-name-of-post.ext
2. Includes the necessary header material (see the next slide)

You can take a look at the source for the existing posts in the _posts folder to get an idea about how it works.

The anatomy of a post

Source: https://raw.githubusercontent.com/MUSA-550-Fall-2020/github-pages-starter/master/_posts/2019-04-17-example-post.md

Rendered: https://musa-550-fall-2020.github.io/github-pages-starter/example-post/

How can we embed our (static) Matplotlib charts?

These are just normal PNG images — we can use Markdown's syntax for embedding images.

Embedding images: the syntax

![alt-text]({{ site.url }}{{ site.baseurl }}/assets/images/YOUR_IMAGE_FILE.png

Steps

1. Place your image in the assets/images/ folder
2. Change YOUR_IMAGE_FILE.png to the name of your image and leave the rest of the path unchanged.

Note: the curly brackets for site.url and site.baseurl are template variables. When the site is rendered, these variables automatically get filled in so that the absolute path to the PNG file is correct.

How can we embed our interactive charts?

In the header, we can specify the charts to load using special loaders, which can handle three different types of files:

• Altair plots via the JSON specification
• Hvplot HTML files
• Folium HTML files

Embedding Altair/Hvplot charts: example

Source: https://raw.githubusercontent.com/MUSA-550-Fall-2020/github-pages-starter/master/_posts/2019-04-13-measles-charts.md

Rendered: https://musa-550-fall-2020.github.io/github-pages-starter/measles-charts/

Embedding Altair/Hvplot charts: the header

Header syntax

Under the altair-loader and hv-loader we need to specify two things:

1. The CSS "id" attribute for the div where we want the chart to go
2. The path to the HTML or JSON file to load (must be in the charts/ folder)

This should be specified as key: value pairs, for example:

altair-chart-1: "charts/measlesAltair.json"

In this case altair-chart-1 is the CSS identifier and "charts/measlesAltair.json" is the name of the file to load.

Note: there must be a matching "div" element with the same CSS identifier — this is where the chart will be embedded!

Important

• For altair charts, you should point the loader the saved JSON representation of the chart
• For hvplot charts, you should point the loader to the saved HTML representation of the chart

Embedding Altair/Hvplot charts: the content

Embedding Folium charts: example

Source: https://raw.githubusercontent.com/MUSA-550-Fall-2020/github-pages-starter/master/_posts/2019-04-13-folium-charts.md

Rendered: https://musa-550-fall-2020.github.io/github-pages-starter/folium-charts/

Embedding Folium charts: the header

Header syntax

For the folium-loader, the syntax is:

CSS identifier for chart : ["chart file path", "width of chart in pixels"]

Again, there must be a matching "div" element with the same CSS identifier — this is where the chart will be embedded!

Embedding Folium charts: the content

So, which files are important?

There's a lot of extra stuff we don't actually need. The most important files/folders are:

• _config.yml: the main configuration file for the site, where you can change names, titles, etc
• _posts/: where the markdown posts (your content!) go
• assets/images/: where you can place your static matplotlib-generated PNG files
• charts/: where you can place your .html or .json files from Altair/Hvplot/Folium

Exercise: Setup your own Github Pages project page

Steps:

• Follow the above instructions to set up your own project website
• Create an interactive Altair chart for the Gapminder health/income dataset
• Create a new post by adding a new post (with the proper file name convention) to the _posts folder
• Embed the Altair chart in your new post

Notes

• As a reminder, the gapminder dataset is available from the built-in vega_datasets package
• We want to plot a scatter chart with the income on the x-axis and the life expectancy on the y-axis

import altair as alt
from vega_datasets import data

source = data.gapminder_health_income.url
source

'https://vega.github.io/vega-datasets/data/gapminder-health-income.csv'

data = pd.read_csv(source)
data.head()

	country	income	health	population
0	Afghanistan	1925	57.63	32526562
1	Albania	10620	76.00	2896679
2	Algeria	13434	76.50	39666519
3	Andorra	46577	84.10	70473
4	Angola	7615	61.00	25021974

Part 3: Web Servers

A few quick basics before we move on to interactive Web apps with Dash...

Some (oversimplified) basics

• The browser needs a file -> it requests it via the HTTP protocol
• The web server receives the request, and if the requested file exists, sends it back to the browser using HTTP

Two main types:

Static and dynamic

Static Web Servers

• Serves hosted HTML files to the browser "as-is"
• Response from the server is always the same — static
• Example: Github Pages

Dynamic Web Servers

• Capable of serving dynamic content to browsers
• Extra software that executes server-side before sending the response back to the browser
• Common example: querying a database server
  • browser sends request for data with specific query parameters
  • server responds dynamically, sending the requested data

For more information: Mozilla documentation

CARTO: a dynamic web server

Anatomy of a request

• Base URL: shown in blue
• ?: separates the base URL from the query parameters
• Query parameters: tells the web server how to respond to the request
• &: separator between the query parameters

Flask: a micro web framework in Python

A very lightweight package for framework for dynamic web apps

Flask documentation

Example 1: your first Flask app

• Store the code for our app in a ".py" file:
• Create a "local" web server by executing our application code locally
• This allows us to build and test our app before deploying the code to a remote web server

Getting set up: launch a terminal in Jupyer notebook

Steps:

From within the Jupyter Notebook Dashboard:

• Navigate to the "hello-flask" folder
• Click on the "hello.py" file to launch the file editor
• Launch a new terminal window by clicking on "New -> Terminal"

Flask code: hello.py

Important: this won't work on Binder, it needs to be running on your local machine

To start a local server

Run the following commands from the Terminal window in Jupyter notebook:

cd hello-flask
python hello.py

The "python" command

• This will execute the *.py file
• Only the code below if __name__ == '__main__': gets executed

StackOverflow answer about the if-name-main syntax in Python.

Note

There are sometimes Anaconda-related issues where "python" will point to the wrong version of Python. You can double check by running:

which python

from the Jupyter Terminal interface. This should print out a path name that includes the name of our Python environment (musa-620)

Take a look at hello.py

View your application!

Navigate in the browser to: http://0.0.0.0:5000

You should see the text: "Hello, World!"

Important: Windows Users

You will likely need to navigate to localhost:5000 instead of http://0.0.0.0:5000

Note: the web address http://0.0.0.0 is an alias for "localhost" — this means the web server is running locally on your computer.

Editing your app with a running server

Try editing the hello.py file in your editor, saving, and re-loading the application page. You should see the app update to reflect the changes!

Example 2: using templates

• Flask can load and render HTML files when specific routes are requested by the browser
• Templates should be stored in a templates/ directory
• The hello-template.py app loads the templates/hello.html when the /hello/ route is requested by the browser

Flask code: hello-template.py

From the terminal window

python hello-template.py

The main page ("/") renders the same thing as the previous app ("Hello, World!").

Navigate to the "/hello/" route and you should see:

Example 3: setting up an API with request parameters

hello-api.py: a more sophisticated example using *request parameters**

Run:

python hello-api.py

Flask code: hello-api.py

Navigate to the "/shootings" route

This uses default request parameters: days = 90 and fatal = 0

Now pass "days" and "fatal" parameters

This returns the number of fatal/nonfatal shootings in the past X "days"

Getting closer to a more realistic use case...

Steps:

1. Get request parameters from the browser (input by the user)
2. Query CARTO database for data based on that input
3. Perform an operation on that data (in this case, count number of shootings)
4. Return computed result back to the browser

Note: If you receive a "OSError: Address already in use"

• If you encounter the "Address already in use" error, you are trying to run multiple Flask servers on the same port
• The port is the number after the address; e.g., for http://0.0.0.0:5000/, the port is 5000
• The easiest solution: run Ctrl-C to end all of the Flask servers but one, or change the port=5000 keyword when initialize the app in the Python script

Now, that we have the basics on to a more advanced library...

Part 4: Dash

The result of Python programmers asking the question: can we build a dashboard just in Python?

The answer is yes...for the most part.

You still need to use some CSS styling and know about different HTML elements. But everything is coded in Python.

Dash

• Built on Flask, similar web framework setup
• Construct Python wrappers of common HTML elements, like "div", "p", etc.
• Allows you to define layout of page purely in Python
• Use Python functions to control how browser interacts with the app — just like Flask

Benefits

• Everything can be done in Python — no more HTML templates
• Use Python to define how user input elements (e.g., sliders) interact with server — no more custom Javascript
• Very sleek, beautiful widgets and interactive components built in to the library
• Can handle Markdown too
• Good tutorials and user guide — https://dash.plot.ly/

Downsides

• Built to be used with the Plotly visualization library
• Private company that has open-sourced their interactive visualization library
• A bit more work to get non-Plotly visualizations working — need to use our IFrame trick

The general workflow

In your Python Dash app, there are two main steps:

1. Define the HTML layout, e.g., "div" elements, slider elements, etc.
2. Define functions that take inputs from the interactive widgets from Step 1. and return the output for other HTML elements.
   • When the user changes a slider, dropdown, etc, the function will run, and return the updated output

Dash HTML components = "the app layout"

• Wrappers for all of the common HTML elements
• Stored in the dash_html_components module

See: https://dash.plot.ly/dash-html-components

# The standard import
import dash_html_components as html

Dash HTML Example

html.Div([
    html.H1('Hello Dash'),
    html.Div([
        html.P('Dash converts Python classes into HTML'),
        html.P('This conversion happens behind the scenes by Dash's JavaScript front-end')
    ])
])

This gets converted automatically to the following HTML in your web app:

<div>
    <h1>Hello Dash</h1>
    <div>
        <p>Dash converts Python classes into HTML</p>
        <p>This conversion happens behind the scenes by Dash's JavaScript front-end</p>
    </div>
</div>

Dash core components = "widgets"

• A set of fancy widgets already built in!
• Stored in the dash_core_components module

See: https://dash.plot.ly/dash-core-components

# The standard import for these components
import dash_core_components as dcc

Note: Markdown is also supported!

Use the dcc.Markdown() object to automatically convert Markdown into HTML in your web apps!

Reference: HTML and CSS tutorials

The core components and HTML components in Dash remove most of the direct HTML/CSS, but still good to know some of the basics:

• HTML Tutorial
• CSS Tutorial

Useful to keep these references handy if you are unsure about syntax for HTML and/or CSS

The Dash Getting Started Guide

https://dash.plot.ly/getting-started

Combining Dash with Altair and Folium

• You are welcome to incorporate Plotly visualizations directly into your dashboard, but it can get complicated quickly
• I am not a huge fan of Plotly's API and documentation
• Focus on a few examples showing Altair + Dash and Folium + Dash

Key: any valid ".html" block can be embedded within an "IFrame" element, as we have seen

Two key elements of Dash apps

• You must define the "app.layout" — this is your HTML layout
• You must define your callback functions and the input/output that they take.
  • You mark certain functions as callbacks so Dash knows they interact with your HTML code.

Steps:

1. Define a IFrame() element as part of the layout
2. Given some user input, generate our Altair charts and save them as HTML
3. Assign the chart HTML to the IFrame on our page

Let's see some Dash + Altair examples...

Two examples in the "dash-altair/" folder...

Visualizing the "penguins" dataset

Just like with Flask, we can run from the Jupyter terminal:

python dash_altair_penguins.py

Dash app

Translating the shootings app to Dash

We can run from the Jupyter terminal:

python dash_altair_shootings.py

Dash app

More information: Dash user guide

• https://dash.plot.ly/getting-started
• Very good series of chapters illustrating different aspects of Dash
• Relies on Plotly visualizations but idea is the same for IFrame elements

One more Dash example: Dash + Folium

• Since Folium plots can be converted to HTML, they can be displayed in Dash apps as HTML IFrame elements
• We can use Dash widgets to control what gets displayed on the map

From the base repository folder, run

cd dash-folium
python app.py

to start the example

Deploying a Dash app

• By default, Dash apps run on "localhost" - you can only access them on your own machine.
• To share a Dash app, you need to "deploy" your Dash app to a server
• Plotly has an "enterprise" (commercial) solution for this but it costs $$$

Recommended solution: Heroku

• Heroku makes deployment of arbitrary apps and dashboards very easy
• Key: Provides a free tier to get you started

Getting started

• First, signup for a Heroku free account
• Download and install the command line interface (CLI).
• Once you are set up follow the instructions to log into the CLI.

Launching your app on Heroku

• The Dash docs contain a complete example of launching a simple Dash app to Heroku
• See the "Heroku Example" section of https://dash.plot.ly/deployment
• Make sure you've signed up for an account and downloaded the CLI first! (see the previous slide)

Summary: web-based visualizations so far

• Github Pages via MUSA-550-Fall-2020/github-pages-starter
  • Supports: Folium, Altair, Hvplot, HTML files via IFrames
  • User cannot use the interactive features of the charts, but cannot change the content dynamically
• Flask/Dash
  • Supports: Folium, Altair, Hvplot, HTML files via IFrames
  • Allows for more complex user interactions via a set of dynamic widgets

Next Week: Panel

• Example gallery: https://panel.holoviz.org/gallery/index.html
• Other demos from a Panel developer: https://jsignell.github.io/

Exercise: modifying the Dash shootings apps

Let's modify either the Altair-based or Folium-based Dash app for the shootings data in Philadelphia to include a dropdown that allows the user to specify which race to show data for.

Steps

• First, take a look at the documentation for the Dropdown object in Dash's core component library: https://dash.plot.ly/dash-core-components/dropdown
• Modify the layout of the app to include a new Div element that contains the Dropdown element
• Modify your render() function to accept the value of the race dropdown as an input to the function — then, you can use that value to select the appropriate subset of your data frame to plot!